Inferring dependencies in Embodiment-based modular reinforcement learning

The state-spaces needed to describe realistic physical embodied agents are extremely large, which presents a serious challenge to classical reinforcement learning schemes. In previous work (Jacob et al., 2005a, Jacob et al., 2005b) we introduced our EMBER (for EMbodiment-Based modulaR) reinforcement learning system, which describes a novel method for decomposing agents into modules based on the agent’s embodiment. This modular decomposition factorises the statespace and dramatically improves performance in unknown and dynamic environments. However, while there are great advantages to be gained from a factorised state-space, the question of dependencies cannot be ignored. We present a development of the work reported in (Jacob et al., 2004) which shows, in a simple example, how dependencies may be identified using a heuristic approach. Results show that the system is able quickly to discover and act upon dependencies, even where they are neither simple nor deterministic.